[JENKINS-63343] Validate element types for collections and maps when deserializing XML files #9727

dwnusbaum · 2024-09-12T19:03:09Z

See JENKINS-63343. This PR checks all fields being deserialized to see if they will be handled by RobustCollectionConverter or RobustMapConverter, and if so, it explicitly constructs an instance of the converter specialized to the type of the field in question so that collection elements and map keys and values can be type checked. Any values which do not have the expected type are ignored, and an OldDataMonitor warning is logged.

Testing done

See new automated tests.

Proposed changelog entries

Ignore values with incorrect types when deserializing collections and maps in XML files.

Proposed upgrade guidelines

N/A

Submitter checklist

Give feedback

The Jira issue, if it exists, is well-described.
The changelog entries and upgrade guidelines are appropriate for the audience affected by the change (users or developers, depending on the change) and are in the imperative mood (see examples). Fill in the Proposed upgrade guidelines section only if there are breaking changes or changes that may require extra steps from users during upgrade.
There is automated testing or an explanation as to why this change has no tests.
New public classes, fields, and methods are annotated with @Restricted or have @since TODO Javadocs, as appropriate.
New deprecations are annotated with @Deprecated(since = "TODO") or @Deprecated(forRemoval = true, since = "TODO"), if applicable.
New or substantially changed JavaScript is not defined inline and does not call eval to ease future introduction of Content Security Policy (CSP) directives (see documentation).
For dependency updates, there are links to external changelogs and, if possible, full differentials.
For new APIs and extension points, there is a link to at least one consumer.
Options

Desired reviewers

@mention

Before the changes are marked as ready-for-merge:

Maintainer checklist

Give feedback

There are at least two (2) approvals for the pull request and no outstanding requests for change.
Conversations in the pull request are over, or it is explicit that a reviewer is not blocking the change.
Changelog entries in the pull request title and/or Proposed changelog entries are accurate, human-readable, and in the imperative mood.
Proper changelog labels are set so that the changelog can be generated automatically.
If the change needs additional upgrade steps from users, the upgrade-guide-needed label is set and there is a Proposed upgrade guidelines section in the pull request title (see example).
If it would make sense to backport the change to LTS, a Jira issue must exist, be a Bug or Improvement, and be labeled as lts-candidate to be considered (see query).
Options

…n some cases

core/src/main/java/hudson/util/RobustCollectionConverter.java

core/src/test/java/hudson/util/RobustCollectionConverterTest.java

dwnusbaum · 2024-09-16T20:21:02Z

core/src/main/java/hudson/util/RobustReflectionConverter.java

+        // TODO: Is there a better way to make something like this work? Does it break any custom converters?
+        if (converter == null && new RobustCollectionConverter(mapper, reflectionProvider).canConvert(type)) {
+            converter = new RobustCollectionConverter(mapper, reflectionProvider, field.getGenericType());
+        }


In particular I am wondering about things like NodeList and AxisList which extend ArrayList. They might be ok because of the exact equality comparisons here, but I am not sure. I guess also that those classes don't support any of the desired recoverability features in RobustCollectionConverter, but that shouldn't be affected by my PR.

I confirmed that NodeList is unaffected by this change - its custom converter continues to be used and its deserialization is not robust against invalid values.

Looking at #342, I think we only expect getLocalConverter to matter for fields that have the @XStreamConverter annotation, which seems to be more or less unused based on this search, and also none of those annotations are for collections or maps.

dwnusbaum · 2024-09-18T21:20:15Z

core/src/main/java/hudson/util/RobustReflectionConverter.java

@@ -451,6 +451,13 @@ private boolean fieldDefinedInClass(Object result, String attrName) {

    protected Object unmarshalField(final UnmarshallingContext context, final Object result, Class type, Field field) {
        Converter converter = mapper.getLocalConverter(field.getDeclaringClass(), field.getName());
+        if (converter == null) {
+            if (new RobustCollectionConverter(mapper, reflectionProvider).canConvert(type)) {


It's a bit awkward to recreate this for every single field being unmarshalled, should we keep an instance as a local field just for these canConvert calls?

dwnusbaum · 2024-09-18T21:23:43Z

core/src/main/java/hudson/util/RobustReflectionConverter.java

@@ -451,6 +451,13 @@ private boolean fieldDefinedInClass(Object result, String attrName) {

    protected Object unmarshalField(final UnmarshallingContext context, final Object result, Class type, Field field) {
        Converter converter = mapper.getLocalConverter(field.getDeclaringClass(), field.getName());
+        if (converter == null) {


See #9727 (comment) for discussion about cases where converter is not null. In short, I don't think we care.

I am not sure though whether this method covers all of the cases we care about. It seems like we may need similar changes related to "implicit collections". I will look into that a bit.

Specifically, prior to this line, I think we should look up the field and then add code comparable to what we added to RobustCollectionConverter to validate the value type:

jenkins/core/src/main/java/hudson/util/RobustReflectionConverter.java

Line 477 in 1836bd6

collection.add(value);

Needs to be investigated with a test case.

I looked into the cases related to implicit collections today. My understanding is that you need to explicitly call addImplicitArray, addImplicitCollection, or addImplicitMap for the relevant code to matter, which is not very common, see here (there are also a few usages in CloudBees plugins). The overloads where you specify the item type already avoid JENKINS-63343 because of getFieldNameForItemTypeAndName here which uses context.getRequiredType() leading to invalid elements being silently ignored. The overloads for arrays and maps seem to not actually work at all due to this check (which leaves me somewhat confused about #2621, but perhaps my testing did not cover all cases).

So, I think we do not really care much about implicit collections, but 09ca355 should handle them too.

…ent types

jglick

Some minor comments, but generally looks good to the extent I understand it; I definitely do not follow all the subtleties about XStream behavior.

core/src/main/java/hudson/util/RobustCollectionConverter.java

core/src/main/java/hudson/util/RobustMapConverter.java

core/src/main/java/hudson/util/RobustReflectionConverter.java

…n tests

dwnusbaum · 2024-10-08T16:38:46Z

/label ready-for-merge

This PR is now ready for merge, after ~24 hours, we will merge it if there's no negative feedback.

Thanks!

[JENKINS-63343] Validate element types in RobustCollectionConverter i…

2f4e04c

…n some cases

jglick reviewed Sep 12, 2024

View reviewed changes

core/src/main/java/hudson/util/RobustCollectionConverter.java Show resolved Hide resolved

jglick reviewed Sep 12, 2024

View reviewed changes

core/src/test/java/hudson/util/RobustCollectionConverterTest.java Outdated Show resolved Hide resolved

dwnusbaum commented Sep 16, 2024

View reviewed changes

dwnusbaum added 3 commits September 18, 2024 16:39

[JENKINS-63343] Also validate keys and values in map entries

966f598

Merge branch 'master' into JENKINS-63343

0944533

[JENKINS-63343] Improve OldDataMonitor error messages and update tests

61db49c

dwnusbaum commented Sep 18, 2024

View reviewed changes

dwnusbaum changed the title ~~[JENKINS-63343] Validate element types in RobustCollectionConverter in some cases~~ [JENKINS-63343] Validate element types for collections and maps during XML deserialization in some cases Sep 23, 2024

dwnusbaum added 2 commits September 23, 2024 14:08

[JENKINS-63343] Also handle implicit collection with unspecified elem…

09ca355

…ent types

[JENKINS-63343] Spotless

db6f157

dwnusbaum changed the title ~~[JENKINS-63343] Validate element types for collections and maps during XML deserialization in some cases~~ [JENKINS-63343] Validate element types for collections and maps during XML deserialization Sep 23, 2024

dwnusbaum changed the title ~~[JENKINS-63343] Validate element types for collections and maps during XML deserialization~~ [JENKINS-63343] Validate element types for collections and maps when deserializing XML files Sep 23, 2024

dwnusbaum added the bug For changelog: Minor bug. Will be listed after features label Sep 23, 2024

dwnusbaum marked this pull request as ready for review September 23, 2024 19:37

Merge branch 'master' into JENKINS-63343

b37ca86

jglick approved these changes Sep 24, 2024

View reviewed changes

dwnusbaum added 5 commits September 24, 2024 11:28

[JENKINS-63343] Avoid rawtypes

c1af611

[JENKINS-63343] Add tests for rawtypes

312e51c

[JENKINS-63343] Adjust Javadoc for clarity

801c3e7

[JENKINS-63343] Avoid ClassCastException and check handling of null i…

df2ac81

…n tests

[JENKINS-63343] Fix Javadoc link by adding import

bc2cf4d

dwnusbaum requested a review from jglick September 24, 2024 17:31

[JENKINS-63343] List.of cannot be used with null values

013bd6e

jglick approved these changes Sep 25, 2024

View reviewed changes

jgreffe approved these changes Sep 25, 2024

View reviewed changes

Merge branch 'master' into JENKINS-63343

c8000fa

comment-ops-bot bot added the ready-for-merge The PR is ready to go, and it will be merged soon if there is no negative feedback label Oct 8, 2024

dwnusbaum merged commit 6651e85 into jenkinsci:master Oct 11, 2024
16 checks passed

dwnusbaum deleted the JENKINS-63343 branch October 11, 2024 18:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[JENKINS-63343] Validate element types for collections and maps when deserializing XML files #9727

[JENKINS-63343] Validate element types for collections and maps when deserializing XML files #9727

dwnusbaum commented Sep 12, 2024 •

edited

Loading

Submitter checklist

Maintainer checklist

dwnusbaum Sep 16, 2024

dwnusbaum Sep 18, 2024

dwnusbaum Sep 18, 2024

dwnusbaum Sep 18, 2024

dwnusbaum Sep 18, 2024

dwnusbaum Sep 18, 2024

dwnusbaum Sep 23, 2024

jglick left a comment

dwnusbaum commented Oct 8, 2024

[JENKINS-63343] Validate element types for collections and maps when deserializing XML files #9727

[JENKINS-63343] Validate element types for collections and maps when deserializing XML files #9727

Conversation

dwnusbaum commented Sep 12, 2024 • edited Loading

Testing done

Proposed changelog entries

Proposed upgrade guidelines

Submitter checklist

Desired reviewers

Maintainer checklist

dwnusbaum Sep 16, 2024

Choose a reason for hiding this comment

dwnusbaum Sep 18, 2024

Choose a reason for hiding this comment

dwnusbaum Sep 18, 2024

Choose a reason for hiding this comment

dwnusbaum Sep 18, 2024

Choose a reason for hiding this comment

dwnusbaum Sep 18, 2024

Choose a reason for hiding this comment

dwnusbaum Sep 18, 2024

Choose a reason for hiding this comment

dwnusbaum Sep 23, 2024

Choose a reason for hiding this comment

jglick left a comment

Choose a reason for hiding this comment

dwnusbaum commented Oct 8, 2024

dwnusbaum commented Sep 12, 2024 •

edited

Loading